Image Encoders Part Deux: Let's ramp things up

Published Sep 21, 2021 by Alex C-G

Let’s get back to how well those image encoders cope with memes. We previously saw that when we were comparing very similar (in terms of pixel value) memes, BigTransfer came out on top. But what if we’re searching variants, like Buff Doge vs Cheems?

As far as I know this meme isn’t in the dataset we’re using. That means it can’t just do a straight up pixel-match like before. So what could happen? My guess here is that:

• CLIP will return doge memes, since it does feature detection that should pick up those chonky doggo features.
• BigTransfer will return stuff with similar pixel values. And since there’s nothing super close, that means it’ll return stuff with a white background, which is what most of the Buff Doge image is.

Okay, fight!

• We’re using the same dataset and index size as last time to ensure it’s a fair fight.
• This time for the sake of brevity and so we don’t lose it over how cute these puppers are, we’ll just check the top three results.

CLIP

Query image	Results

BigTransfer

Query image	Top 3 results

Oddly, BigTransfer surfaced this as the fifth result, so maaaybe it’s recognizing something like the dogs? Or maybe just a fluke.

So, was I right?

Nope, off by a long shot. Perhaps there’s something weird going on with CLIP’s feature detection. Either way, that’s two for two for BigTransfer when it comes to searching memes. Though it might’ve just got lucky hitting that fifth image.

In short, both encoders suck for this kind of thing, at least out of the box.

What’s next?

I’m exhausted with memes for now. But if anyone has any ideas for what to try next (and preferably wants to take part!) ping me on Twitter at @alexcg.